The outcome of a statistical test is a decision to either accept the

, or reject

in favor of

.

Because

pertains to the population’s true value, the effect you see in your sample is either true or

false for the population from which you are sampling. You may never know what that truth is, but an

objective truth is out there nonetheless.

The truth can be one of two answers — the effect is there, or the effect is not there. Also, your

conclusion from your sample will be one of two answers — the effect is there, or the effect is not

there.

These factors can be combined into the following four situations:

Your test is not statistically significant, and H0is true. This is an ideal situation because the

conclusion of your test matches the truth. If you were testing the mean difference in effect between

Drug A and Drug B, and in truth there was no difference in effect, if your test also was not

statistically significant, this would be an ideal result.

Your test is not statistically significant, but H0is false. In this situation, the interpretation of your

test is wrong and does not match truth. Imagine testing the difference in effect between Drug C and

Drug D, where in truth, Drug C had more effect than Drug D. If your test was not statistically

significant, it would be the wrong result. This situation is called Type II error. The probability of

making a Type II error is represented by the Greek letter beta (β).

Your test is statistically significant, and HAltis true. This is another situation where you have an

ideal result. Imagine we are testing the difference in effect between Drug C and Drug D, where in

truth, Drug C has more of an effect. If the test was statistically significant, the interpretation would

be to reject H0, which would be correct.

Your test is statistically significant, but HAltis false. This is another situation where your test

interpretation does not match the truth. If there was in truth no difference in effect between Drug A

and Drug B, but your test was statistically significant, it would be incorrect. This situation is

called Type I error. The probability of making a Type I error is represented by the Greek letter

alpha (α).

We discussed setting α = 0.05, meaning that you are willing to tolerate a Type I error rate of 5

percent. Theoretically, you could change this number. You can increase your chance of making a

Type I error by increasing your α from 0.05 to a higher number like 0.10, which is done in rare

situations. But if you reduce your α to number smaller than 0.05 — like 0.01, or 0.001 — then you

run the risk of never calculating a test statistic with a p value that is statistically significant, even

if a true effect is present. If α is set too low, it means you are being very picky about accepting a

true effect suggested by the statistics in your sample. If a drug really is effective, you want to get a

result that you interpret as statistically significant when you test it. What this shows is that you

need to strike a balance between the likelihood of committing Type I and Type II errors —

between the α and β error rates. If you make α too small, β will become too large, and vice versa.